Extending MSBuild

MSBuild is the build platform for Microsoft and Visual Studio. Unlike other build systems, for example NAnt, MSBuild not only provides you with a scripting language for controlling builds, but also with a default build process that can be extended and altered to make it suit your needs.

The benefits of MSBuild, over other build tools, are time savings, code base reduction, harnessing of tried and tested Microsoft code (don’t laugh), and even one or two unique features (assuming that you are using .NET, of course!).

In this article I will first explore the default MSBuild process and then show how it can be altered by modifying and overwriting predefined properties and items, and by inserting your own custom targets. The overall aim is to promote an understanding of the default build process that will enable you to discover your own ways of extending MSBuild.

NOTE:
This article will refer to .NET 2.0, but as far as I know everything applies equally to .NET 3.5. The article requires some prior knowledge of MSBuild. See here for an overview from the horse’s mouth: MSDN: Visual Studio MSBuild Concepts.

Project Files are MSBuild Files

There are two important points to notice when working with MSBuild.

An important consequence of the first point is that not only can you customise MSBuild command line builds, but also builds that are started from within Visual Studio. In .NET there is no difference between the two. You can distinguish between command line and Visual Studio builds by querying the BuildingInsideVisualStudio property.

The Default Build Process

Let’s start from the top. You can build any Visual Studio project by executing a command line like so:

msbuild app.csproj

Notice that we didn’t specify a target, i.e. when MSBuild runs; it will call the default target of app.csproj.

Now, let’s look at a project file generated by Visual Studio. Below is the project file of a freshly created C# class library (with some omissions).

<Project DefaultTargets=”Build”
xmlns=”http://schemas.microsoft.com/developer/msbuild/2003″>

  <PropertyGroup>

    <Configuration Condition=” ‘$(Configuration)’ == ” “>Debug</Configuration>

    <Platform Condition=” ‘$(Platform)’ == ” “>AnyCPU</Platform>

    <ProductVersion>8.0.50727</ProductVersion>

    <SchemaVersion>2.0</SchemaVersion>

    <ProjectGuid>{5B9EEF3B-7CF4-4D38-B80D-E07F4B1E3CD0}</ProjectGuid>

    <OutputType>Library</OutputType>

    <AppDesignerFolder>Properties</AppDesignerFolder>

    <RootNamespace>ClassLibrary1</RootNamespace>

    <AssemblyName>ClassLibrary1</AssemblyName>

  </PropertyGroup>

<PropertyGroup Condition=” ‘$(Configuration)|$(Platform)’ == ‘Debug|AnyCPU’ “>

  …

  </PropertyGroup>

  <PropertyGroup Condition=” ‘$(Configuration)|$(Platform)’ == ‘Release|AnyCPU’ “>

    …

  </PropertyGroup>

  <ItemGroup>

    <Reference Include=”System” />

    <Reference Include=”System.Data” />

    <Reference Include=”System.Xml” />

  </ItemGroup>

  <ItemGroup>

    <Compile Include=”Class1.cs” />

    <Compile Include=”Properties\AssemblyInfo.cs” />

  </ItemGroup>

  <Import Project=”$(MSBuildBinPath)\Microsoft.CSharp.targets” />

</Project>

As you can see, there is not a lot in it. There are some properties which you might recognise from the Properties pages of your projects. Furthermore, there are two item groups, one called ‘Reference’, containing the references, and another called ‘Compile’, containing the source files.

So where is the action? In particular, where is the default ‘Build’ target that’s called when Visual Studio builds the project?

The answer is of course this line:

   <Import Project=”$(MSBuildBinPath)\Microsoft.CSharp.targets” />

‘MSBuildBinPath‘ is a reserved property and evaluates to the path where MSBuild.exe resides, i.e. usually something like C:\WINNT\Microsoft.NET\Framework\v2.0.50727. (See here for a list of reserved properties http://msdn2.microsoft.com/en-us/library/0k6kkbsd(VS.80).aspx )

However, perusing Microsoft.CSharp.targets reveals that there isn’t much in there either, appart from a target called ‘CoreCompile’ which calls the Csc task. In particular, there is no ‘Build’ target here either.

However, at the bottom you’ll find another import,

<Import Project=”Microsoft.Common.targets” />

This is where all the action is. Microsoft.Common.targets contains the ‘Build’ target we were looking for, and most of the targets, properties and items executed during a build are defined here.

Property, Item and Target Evaluation

When hooking into the default build process you need to know how MSBuild evaluates properties, items and targets. In particular, note that:

a) Properties, items and targets are evaluated in order from top to bottom. Furthermore, properties, items and targets that appear later in the build script always overwrite those of the same name that appear earlier in the build script.

b) Properties and items defined in property and item groups are evaluated statically. What that means is that all properties and items, no matter where they appear in the build script, are evaluated at the start, before any targets have executed.

This implies that, most of the time, you will want to make your additions after the default build process has been declared, that is in your project file below the Microsoft.CSharp.targets import. This will allow you to use properties and items defined by Microsoft, to overwrite and amend those properties and items, and to overwrite and extend predefined targets. Moreover, even though you modify properties and items at the bottom of your build script, these modifications will have been evaluated by the time the Microsoft targets run. You can thus provide custom values to default targets. (See the section below, ‘Referencing Different Dlls for Release and Debug Builds’, for an example.)

This also means that you must be very careful not to modify or overwrite existing properties, items and targets when you don’t intend to. To avoid conflicts, I suggest always adding a unique pre-fix to your custom property, item and target names, for example your company name followed by an underscore.

Other points to note about property evaluation are, that a) property values passed in via the command line always overwrite property values set in build scripts, and b) all environment variables are automatically available as properties in your build scripts. For example, ‘$(PATH)‘ evaluates to the value of the PATH environment variable. Properties defined in scripts overwrite properties defined by environment variables, however.

It is possible to not overwrite earlier properties by using the following idiom:

  <Property Condition=” ‘$(Property)’ == ” “>Value</Property>

This will only set the value of Property to Value if property hasn’t been assigned a value previously. Exercise care though, as this does not work for properties for which the empty string is a valid value.

Side Effects of Static Evaluation

One unfortunate side-effect of static item evaluation manifests itself when specifying items with wild-cards. Imagine writing the following script,

  <ItemGroup>

    <OutputFile Include=”$(OutputPath)\**\*” />

</ItemGroup>

  <Target Name=”CopyOutputFiles”>

    <Copy SourceFiles=”OutputFile” DestinationFolder=”$(DestFolder)” />

  </Target>

On the face of it, we are defining an item which contains all output files. We then use this item to copy the output files to another folder. If you run the script, however, you will find that not a single file was copied. The reason is that, due to static evaluation, the item got evaluated before any targets ran. In other words, before any output was generated. The item is thus empty.

Dynamic items get around this problem. The example would then look like this,

  <Target Name=”CopyOutputFiles”>

    <CreateItem Include=”$(OutputPath)\**\*”>

      <Output TaskParameter=”Include” ItemName=”OutputFile”/>

    </CreateItem>

    <Copy SourceFiles=” @(OutputFile)” DestinationFolder=”$(DestFolder)” />

</Target>

Now the item is created just before the Copy task is called and after all the output creating targets have run. Sadly, dynamic items and properties are not very nice, in that they require an excessive amount of difficult-to-read code.

Discovering Predefined Properties and Items

The default build process defines a whole raft of properties and items that you can use for your own purposes. The three I found most useful were:

• $(Configuration) – The configuration you are building, i.e. either ‘Debug’ or ‘Release’
• $(OutputPath) – The output path as defined by your project
• @(MainAssembly) – The main assembly generated by your project

There are two ways to discover what other properties and items are available. One is by inspecting the Microsoft build scripts. This can be tedious if you don’t know exactly what you are looking for, but it is useful if you know the target whose properties and items you want to modify. In the latter case, looking at the target should tell you exactly what properties and items are of interest.

The second way is to set verbosity to diagnostic on a command line build:

msbuild app.csproj /verbosity:diagnostic

This will result in all defined properties and items, together with their values, being listed at the top of the output. From there it’s easy to pick the ones you need. Bear in mind, though, that dynamic properties and items will not be listed. For those it’s back to scanning the Microsoft build scripts, I’m afraid.

Diagnostic verbosity is also useful for debugging your scripts, in general. It gives very detailed output, as well as the aggregate timings of each task and target.

When running from the command line, I recommend redirecting output to a file, as printing to the screen can slow down execution considerably. That way you also get the complete output, and you can search it in a text editor.

Referencing Different Dlls for Release and Debug Builds

Let’s use all the above in an example. Assume we want to reference different dlls, depending on whether we are building a debug or a release build. First, we need two folders that hold the different dlls, ‘Debug’ and ‘Release’, say. And let’s assume they reside in the same folder as the .csproj file.

Next, from reading the documentation and inspecting previous build output, I know what I want to hook into is the call to the ResolveAssemblyReference task. This task can be found in Microsoft.Common.targets in the ResolveAssemblyReferences target, and looks like this (when you squint a little),

        <ResolveAssemblyReference

            …

            SearchPaths=”$(AssemblySearchPaths)”

            …

        </ResolveAssemblyReference>

So, all we need to do is extend the AssemblySearchPaths property. To do so, add the following to the bottom of your project file,

  <PropertyGroup>

<AssemblySearchPaths>

  $(Configuration);

      $(AssemblySearchPaths)

    </AssemblySearchPaths>

  </PropertyGroup>

What we’re doing here is defining a new property, also called AssemblySearchPaths. Since our definition occurs below the original definition we overwrite the original definition. The new definition states that AssemblySearchPaths consists of the value of Configuration followed by whatever the old value of AssemblySearchPaths was. In effect, we prepend the value of Configuration to the value of the AssemblySearchPaths property.

Now, when the ResolveAssemblyReference task runs, it will use our new definition of AssemblySearchPaths, thanks to static evaluation. It will look for referenced dlls in a folder, called the value of the Configuration property, before it looks anywhere else. In the case where you are building a Debug build it would look first in a subfolder of the current folder called ‘Debug’. Since the current folder is always the folder in which your ‘start-up project’ resides, i.e. the project file, we are done.

The changed value of the AssemblySearchPaths property can be verified by looking at the build output with verbosity set to diagnostic.

The cool thing is that this change takes effect even when building inside Visual Studio. In other words, when you set the configuration drop down to ‘Release’ you are referencing a release build and when you set it to ‘Debug’ a debug build.

Executing Custom Targets

Most of the time, when extending MSBuild, you will want to insert your own custom targets. There are two ways to go about this. The first is to overwrite predefined targets with your own. These targets are defined in Microsoft.Common.targets, like so,

<Target Name=”BeforeBuild”/>

As you can see, the above target does nothing. However, it gets called during each build at the approprieate time, i.e. before the build starts. So, if you now define a target of the same name, your target will overwrite the one in Microsoft.Common.targets, and your target will be called instead. There is a list of available targets here, http://msdn2.microsoft.com/en-us/library/ms366724(VS.80).aspx.

The second method for inserting your own targets into the build process is to modify the ‘DependsOn‘ properties. Most targets in Microsoft.Common.targets have the DependsOnTargets attribute set to be the value of a property, whose name is of the form ‘xxxDependsOn‘. Where ‘xxx‘ is the name of the target. For example, the ‘Build’ target depends on whatever the value of the BuildDependsOn property is.

    <Target

        Name=”Build”

        DependsOnTargets=”$(BuildDependsOn)”/>

To insert your own target, you have to modify the value of the BuildDependsOn property, like so,

    <PropertyGroup>

        <BuildDependsOn>

            CustomTargetBefore;

            $(BuildDependsOn);

            CustomTargetAfter

        </BuildDependsOn>

    </PropertyGroup>

The outcome of this will be that CustomTargetBefore will run before all previously defined targets in BuildDependsOn, and CustomTargetAfter will run after all previously defined targets. The advantage of using the ‘DependsOn’ properties is, that you don’t inadvertently overwrite somebody elses targets, as is possible when overwriting predefined targets.

Extending All Builds

Occasionally, you will want to make extensions to the build process that apply to all builds. For example, if you have a dedicated build server and you want to automatically obfuscate each build. One way to do this would be to import a common file into all your project files. There is, however, the danger that someday someone forgets to import it and, without any warning, you would end up with unobfuscated builds. As an alternative, you could modify Microsoft.Common.targets et al. This is also not desirable. For example, if a future Framework installation modifies those files and removes your additions you would again end up with unobfuscated builds, and no warning.

Luckily Microsoft has anticipated this scenario and allows you to specify custom build files. If these custom files exist, they are imported by Microsoft.Common.targets on every build. From those files you can make the same modifications to the default build process as discussed previously, but they will apply to every build.

There are two files available:

1. Custom.Before.Microsoft.Common.targets which is imported at the top of Microsoft.Common.targets
2. Custom.After.Microsoft.Common.targets which is imported at the bottom.

Both files are expected to reside in %program files%\MSBuild\v2.0. However, since these files are defined via the properties CustomBeforeMicrosoftCommonTargets and CustomAfterMicrosoftCommonTargets, you can supply your own file names via the command line, like so,

msbuild.exe app.proj /property:CustomAfterMicrosoftCommonTargets=custom.proj

Most of the time you will want to use Custom.After.Microsoft.Common.targets, so that you can overwrite and extend existing properties, items and targets, as you would in your project file.

Coming back to the obfuscation example, you would have to create a project file called ‘Custom.After.Microsoft.Common.targets‘ and put it in %program files%\MSBuild\v2.0. In that file you would have to define a target and hook it in like so,

  <PropertyGroup>

    <BuildDepensOn>

      $(BuildDependsOn);

      Obfuscate

    </BuildDepensOn>

  </PropertyGroup>

  <Target Name=”Obfuscate”>

    …

  </Target>

MSBuild Constrictions

Having evangelised, at some length, the wonderful ways in which one can extend MSBuild, I can’t help but mention two serious flaws. The first problem is that Visual Studio solution files are not MSBuild files. MSBuild can, and does execute solution files, but these files are not in the MSBuild format and hence cannot be extended in the same way as project files. What actually happens when MSBuild is called with a solution file, is that it converts the solution file to an MSBuild file in memory and then runs that file. You can even save that file to disk and then extend it when running command line builds. You cannot, however, make extensions at the solution level, that take effect when building inside Visual Studio. Also, when you change your solution file you will have to merge the changes into any generated ‘MSBuild Solution’.

The second problem concerns managed C++ projects. And yes, you guessed it, they are not MSBuild projects either. They are VCBuild projects. When MSBuild builds a managed C++ project it simply calls VCBuild and passes it the project. From there, it gets very tricky and labour intensive to integrate managed C++ projects into your build system. So from a build manager’s point of view it’s good advice to steer clear of managed C++.

Conclusion

If you’re using Visual Studio, you are already using MSBuild, because Visual Studio calls MSBuild whenver you click the ‘Build’ button. The Visual Studio integration of MSBuild allows you to extend your local builds in lots of wonderful ways. In addition, if you’re using .NET, using ‘extended MSBuild’ for your automated builds is a good idea, given the savings in time, effort and code. I hope to have shown how you can explore and extend the default build process to make it suit your needs.